Repository navigation
Conversation
…versions InferenceX collectors leave rows that break the result row contract out of results_<prefix> and eval_results_<prefix> and list them in rejected_rows_* artifacts (SemiAnalysisAI/InferenceX#3806). The per-config bmk_* and eval_* artifacts still carry those rows, and the ingest reads them directly. CI and GCS ingests now read the rejected_rows lists before writing. They skip a per-config benchmark file whose <artifact>/<file> path and content match an entry, and a per-config eval task whose results file and task match one. An unreadable or unidentifiable list fails the ingest instead of being ignored. Benchmark, aggregate eval, and run-stats rows stamped with a result_schema_version other than 1 are refused. Unstamped rows from current InferenceX main and historical artifacts ingest as before. Both skips are counted by the skip tracker and printed in the ingest summary.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to SemiAnalysisAI/InferenceX#3806 and #1293.
InferenceX#3806 makes its collectors leave rows that break the result row contract out of
results_<prefix>andeval_results_<prefix>, list them inrejected_rows_*artifacts, and stamp published rows withresult_schema_version: 1. This ingest also reads the per-configbmk_*andeval_*artifacts, which still contain the quarantined rows, so those rows could reach the database.Changes
ingest-ci-run.ts) and the GCS restore (ingest-gcs-backup.ts) readrejected_rows_<prefix>andrejected_rows_eval_<prefix>before any write. Both download paths already fetch every artifact, soprepare-ci-artifacts.tsis unchanged.<artifact>/<file>path equals an entry'ssourceand its parsed content equals the entry'srow. The content check keeps a list left by an earlier attempt from hiding a file that a later attempt re-uploaded under the same name.row.sourceandrow.task. Eval sources start with the collector's download directory (eval_results/), so only the last two segments,<artifact>/<file>, are compared.rejected_rows_run_statsis not read becauserun-statsalready omits those entries.mapBenchmarkRow,mapAggEvalRow, and a new sharedmapRunStatsrefuse anyresult_schema_versionother than1. Unstamped rows from current InferenceX main, historical artifacts, and GCS backfills ingest as before.unsupported result_schema_versionandquarantined by InferenceX.docs/data-pipeline.mddescribes the contract handling.InferenceX change
Python wrote non-finite floats in
rejected_rows.jsonas bareNaNorInfinitytokens, whichJSON.parserejects, and a NaN row is one of the contract's own rejection cases. SemiAnalysisAI/InferenceX@72e9b2334 on #3806 writes them as strings such as"NaN".#1293 keeps
result_schema_version: 1out of benchmark metrics. It is independent of this PR, and both are needed.Checks
bun run typecheck,bun run lint,bun run fmt, andbun run test:unitpass locally.